Network memory architecture

ABSTRACT

A network memory system comprises a first appliance and a second appliance. The first appliance receives data and determines whether a portion of the data is locally accessible to the second appliance. The first appliance generates an instruction based on the determination and transfers the instruction to the second appliance over a communication network. The second appliance receives the instruction from the first appliance over the communication network and processes the instruction to obtain the data. The second appliance then transfers the data to a computer.

BACKGROUND

1. Technical Field

The present invention relates generally to network memory and moreparticularly to a network memory architecture.

2. Description of Related Art

To allow remote employees access to an enterprise's information systems,organizations typically choose between two networking approaches:centralized servers or distributed servers. Centralized serverimplementations have the advantage of simplicity since an informationtechnology (IT) professional centrally manages, maintains, and enforcespolicies for the organization's data.

FIG. 1 illustrates a centralized server system 100 in the prior art. Thecentralized server system 100 includes a branch office 110 and a centraloffice 120 coupled by a communication network 130. The communicationnetwork 130 forms a WAN between the branch office 110 and the centraloffice 120.

Typically, the central servers 160 in the central office 120 store theorganization's data. Computers 140 make requests for the data from thecentral servers 160 over the communication network 130. The centralservers 160 then return the data to the computers 140 over thecommunication network 130.

The communication network 130 typically comprises a private network(e.g., a leased line network) or a public network (e.g., the Internet).The connections to the communication network 130 from the branch office110 and the central office 120 typically cause a bandwidth bottleneckfor exchanging the data over the communication network 130. The exchangeof the data between the branch office 110 and the central office 120, inthe aggregate, will usually be limited to the bandwidth of the slowestlink in the communication network 130.

For example, the router 150 connects to the communication network 130 bya T1 line, which provides a bandwidth of approximately 1.544Megabits/second (Mbps). The router 170 connects to the communicationnetwork 130 by a T3 line, which provides a bandwidth of approximately 45Megabits/second (Mbps). Even though the communication network 130 mayprovide an internal bandwidth greater than 1.544 Mbps or 45 Mbps, theavailable bandwidth between the branch office 110 and the central office120 is limited to the bandwidth of 1.544 Mbps (i.e., the T1 connection).Connections with higher bandwidth to relieve the bandwidth bottleneckacross the communication network 130 are available, but are generallyexpensive and have limited availability.

Moreover, many applications do not perform well over the communicationnetwork 130 due to the limited available bandwidth. Developers generallyoptimize the applications for performance over a local area network(LAN) which typically provides a bandwidth between 10 Mbps toGigabit/second (Gbps) speeds. The developers of the applications assumesmall latency and high bandwidth across the LAN between the applicationsand the data. However, the latency across the communication network 130typically will be 100 times that across the LAN, and the bandwidth ofthe communication network 130 will be 1/100th of the LAN.

Alternatively, many organizations select the distributed serverimplementation to mitigate some of the problems with the centralizedserver implementation. FIG. 2 illustrates a distributed server system200 in the prior art. The distributed server system 200 includes abranch office 210, a central office 220, and a communication network230. The communication network 230 forms a WAN between the branch office210 and the central office 220.

In the distributed server system 200, the branch servers 240 (e.g.,email servers, file servers and databases) are placed locally in thebranch office 210, rather than solely in the central office 220. Thebranch servers 240 typically store all or part of the organization'sdata. The branch servers 240 generally provide improved applicationperformance and data access. The branch servers 240 respond to a requestfor the organization's data from the local data. For each request forthe data, the central servers 270 potentially do not need to transferthe data over the communication network 130 (i.e., the WAN).Synchronization and backup procedures are implemented to maintain thecoherency between the local data in the branch office 210 and the datain the central office 220.

Unfortunately, managing the distributed server system 200 is complex andcostly. From a physical point of view, the distributed server system 200with one hundred branch offices requires an order of one hundred timesmore equipment than the centralized server approach. Each piece of theequipment not only needs to be purchased, but also installed, managed,and repaired driving significant life cycle costs. The branch office 210may need additional local IT personnel to perform operations because ofthis “Server Sprawl”. Furthermore, the multiplication of managed devicesmeans additional license costs, security vulnerabilities, and patchingactivities.

In distributed server implementations (e.g., the distributed serversystem 200), the data, including the “golden copy” or most up-to-dateversion of mission critical data, is often stored (at least temporarily)only on the branch servers 240 in the branch office 210. Organizationsimplement complex protocols and procedures for replication andsynchronization to ensure that the mission critical data is backed upand kept in-sync across the WAN with the central servers 270.

Furthermore, although FIG. 1 and FIG. 2 illustrate a single branchoffice and a single central office, multiple branch offices and multiplecentral offices exacerbate the previously discussed problems. Forexample, in a centralized server implementation having multiplebranches, computers in each of the multiple branch offices make requestsover the WAN to central servers for the organization's data. The datatransmitted by the central servers in response to the requests quicklysaturate the available bandwidth of the central office's connection tothe communication network, further decreasing application performanceand data access at the multiple branch offices. In a distributed serverimplementation having multiple branches, the cost to provide branchservers in each of the multiple branch offices increases, as well as theproblems of licensing, security vulnerabilities, patching activities,and data replication and synchronization. Moreover, different branchesmay simultaneously attempt to modify the same piece of information.Maintaining coherency in a distributed implementation requires complexand error prone protocols.

As well as implementing centralized servers or distributed servers,organizations also implement mechanisms for caching to improveapplication performance and data access. A cache is generally used toreduce the latency of the communication network (e.g., communicationnetwork 230) forming the WAN (i.e., because the request is satisfiedfrom the local cache) and to reduce network traffic over the WAN (i.e.,because responses are local, the amount of bandwidth used is reduced).

Web caching, for example, is the caching of web documents (i.e., HTMLpages, images, etc.) in order to reduce web site access times andbandwidth usage. Web caching typically stores local copies of therequested web documents. The web cache satisfies subsequent requests forthe web documents if the requests meet certain predetermined conditions.

One problem with web caching is that the web cache is typically onlyeffective for rarely modified static web documents. For dynamicdocuments, there is a difficult trade off between minimizing networktraffic and the risk of the web cache serving up stale data. The webcache may serve stale data because the web cache responds to requestswithout consulting the server.

Another problem is that the web cache does not recognize that twootherwise identical documents are the same if they have differentUniform Resource Locator (URL). The web cache does not consider thecontent or context of the documents. Thus, the web cache caches thedocuments by URL or filename without a determination of the content orcontext of the document. Moreover, the web cache stores entire objects(such as documents) and cache-hits are binary: either a perfect match ora miss. Even where only small changes are made to the documents, the webcache does not use the cached copy of the documents to reduce networktraffic.

SUMMARY OF THE INVENTION

The invention addresses the above problems by providing a network memorysystem and method implemented by the system. A network memory systemcomprises a first appliance and a second appliance. The first appliancereceives data and determines whether a portion of the data is locallyaccessible to the second appliance. The first appliance generates aninstruction based on the determination and transfers the instruction tothe second appliance over a communication network. The second appliancereceives the instruction from the first appliance over the communicationnetwork and processes the instruction to obtain the data. The secondappliance then transfers the data to a computer.

Advantageously, the first appliance may not transfer the data over thecommunication network if the data is locally accessible to the secondappliance. Based on the instruction, the second appliance obtains thedata and transfers the data to the computer. In one embodiment, thecommunication network may comprise a wide area network (WAN). Thenetwork memory system effectively reduces latency over the communicationnetwork, and reduces network traffic by minimizing the amount of datasent over the communication network. In a further advantage, the firstappliance may assume the role of the second appliance, and the secondappliance may assume the role of the first appliance, thereby reducingnetwork traffic and latency for uni-directional and bi-directionalcommunication over the communication network. The second appliancereceives the data and determines whether a portion of the data islocally accessible to the first appliance. The second appliancegenerates another instruction based on the determination and transferthe another instruction to the first appliance over the communicationnetwork. The first appliance receives the another instruction from thesecond appliance over the communication network and process the anotherinstruction to obtain the data. The first appliance then transfers thedata to another computer.

In some embodiments, the first appliance generates the instructionindicating to store the data in a database. The instruction may alsoindicate to retrieve the data from the database. In some embodiments,the first appliance may generate a plurality of instructions. Theplurality of instructions may indicate a plurality of indexes forretrieving the data. The instruction may also indicate an index withinthe database for the data. Accordingly, the second appliance may storethe data in the database based on the instruction. The second appliancemay retrieve the data in the database based on the instruction.

The first appliance may transfer to the second appliance a portion ofthe data that is not locally accessible. The second appliance receivesthe transferred portion of the data, retrieves the portion of the datathat is locally accessible to the second appliance, and processes theportions to obtain the data. The first appliance may receive the datafrom a computer server. The second appliance may also transfer the datato the computer based on a request for the data over the communicationnetwork by the computer. The network memory system therefore providesthe advantages and simplicity of centralized data storage, with theability to quickly retrieve locally accessible data.

Further, in some embodiments, the first appliance further determineswhether the data is locally accessible to a third appliance. The firstappliance then generates another instruction to the third appliancebased on the determination and transfers the another instruction to thethird appliance over the communication network. The third appliancereceives the another instruction from the first appliance over thecommunication network and processes the another instructions to obtainthe data. The third appliance then transfers the data to anothercomputer. Furthermore, the network memory system may localize a copy ofthe data in multiple appliance/node implementations without the licensefees and expenses, maintenance, and security costs of distributed serverhardware.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a centralized server system in the prior art;

FIG. 2 illustrates a distributed server system in the prior art;

FIG. 3 illustrates a network memory system, in an exemplaryimplementation of the invention;

FIG. 4 illustrates a message sequence chart for the network memorysystem where a response to a data request is not locally accessible to abranch appliance, in an exemplary implementation of the invention;

FIG. 5 illustrates data structures for the network memory system todetermine whether a portion of the data is locally accessible to thebranch appliance, in an exemplary implementation of the invention;

FIG. 6 illustrates a message sequence chart for the network memorysystem where the response to the data request is locally accessible tothe branch appliance, in an exemplary implementation of the invention;

FIG. 7A and FIG. 7B illustrate a message sequence chart for the networkmemory system where a portion of the response to the data request islocally accessible to the branch appliance, in an exemplaryimplementation of the invention;

FIG. 8 illustrates a block diagram of the branch appliance, in anexemplary implementation of the invention;

FIG. 9 illustrates a block diagram of a central appliance, in anexemplary implementation of the invention;

FIG. 10 illustrates a network memory system between a first office, asecond office, and a third office, in an exemplary implementation of theinvention; and

FIG. 11 illustrates a message sequence chart for the network memorysystem for discovery and reconciliation, in an exemplary implementationof the invention.

DETAILED DESCRIPTION OF THE INVENTION

The embodiments discussed herein are illustrative of one example of thepresent invention. As these embodiments of the present invention aredescribed with reference to illustrations, various modifications oradaptations of the methods and/or specific structures described maybecome apparent to those skilled in the art. All such modifications,adaptations, or variations that rely upon the teachings of the presentinvention, and through which these teachings have advanced the art, areconsidered to be within the scope of the present invention. Hence, thesedescriptions and drawings should not be considered in a limiting sense,as it is understood that the present invention is in no way limited toonly the embodiments illustrated.

To provide improved application performance and data access, the networkmemory system generally comprises a first appliance and a secondappliance. The first appliance receives data and determines whether aportion of the data is locally accessible to the second appliance. Thefirst appliance generates an instruction based on the determination andtransfers the instruction to the second appliance through thecommunication network.

The network memory system provides that the second appliance processesthe instruction to obtain the data and transfers the data to a computer.The data may be locally accessible to the second appliance, and thetransfer to the computer may occur faster than transferring the dataover the communication network. Accordingly, the second appliancetransfers the data to computer without the first appliance transferringthe data over the communication network that may have a high latency andlow bandwidth. Thus, the network memory system operates to reducelatency and network traffic over the communication network.

FIG. 3 illustrates a network memory system 300, in an exemplaryimplementation of the invention. The network memory system 300 includesa branch office 310, a central office 320, and a communication network330. The branch office 310 includes computers 340, a branch appliance350, and a router 360. The central office 320 includes central servers370, a central appliance 380, and a router 390.

In the branch office 310, the computers 340 are linked to the branchappliance 350. The branch appliance 350 is linked to the router 360. Therouter 360 is coupled to the communication network 330. In the centraloffice 320, the central servers 370 are linked to the central appliance380. The central appliance 380 is linked to the router 390. The router390 is coupled to the communication network 330.

The principles discussed herein are equally applicable to multiplebranch offices (not shown) and to multiple central offices (not shown).For example, the network memory system 300 may include multiple branchoffices and/or multiple central offices coupled to the communicationnetwork 330. Branch office/branch office communication and centraloffice/central office communication, as well as multi-appliance and/ormulti-node communication and bi-directional communication are furtherwithin the scope of the disclosure. However, for the sake of simplicity,the disclosure illustrates the network memory system 300 having thesingle branch office 310 and the single central office 320, and therespective branch office 310/central office 320 communication.

The communication network 330 comprises hardware and/or softwareelements that enable the exchange of information (e.g., voice and data)between the branch office 310 and the central office 320. Some examplesof the communication network 330 are a private wide-area network (WAN),and the Internet. Typically connections from the branch office 310 tothe communication network 330 (e.g., from the router 360 and the router390) are ISDN, T1 lines (1.544 Mbps), and possibly broadband connectionssuch as digital subscriber lines (DSL) and cable modems. Other examplesare T3 lines (43.232 Mbps), OC3 (155 Mbps), and OC48 (2.5 Gbps),although more costly and more likely used for interconnection at thecentral office 320 or as the backbone of the communication network 330.

The branch appliance 350 comprises hardware and/or software elementsconfigured to receive data (e.g., email, files, and databasestransactions), determine whether a portion of the data is locallyaccessible to an appliance (e.g., the central appliance 380), generatean instruction based on the determination, and transfer the instructionto the appliance. The branch appliance 350 also comprises hardwareand/or software elements configured to receive an instruction from anappliance (e.g., the central appliance 380), process the instruction toobtain data, and transfer the data to a computer (e.g., the computers340). One example of the branch appliance 350 is described below withrespect to FIG. 8. The operations of the branch appliance 350 arediscussed in further detail below in FIGS. 4, 5, 6, and 7A-7B.

Locally accessible data comprises any data transferable to the computer(e.g., the computers 340 and the central servers 370) by an appliance(e.g., the branch appliance 350 and the central appliance 380) withouttransferring the data over the communication network 330. In someexamples, the locally accessible data is stored in random access memory(RAM) in the branch appliance 350, on a hard drive in the branchappliance 350, and a combination of data stored in RAM and on one ormore hard drives in the branch appliance 350. In another example, thelocally accessible data is accessible by the branch appliance 350 over acommunication network (other than the communication network 330), suchas data stored in a network attached storage (NAS) device that isinternal or external to the branch office 310. In still another example,the locally accessible data is stored in a database. The database may bestored in RAM, on a hard disk, a combination of RAM and hard disks, in aNAS device, and/or in other optical and flash storage devices.

The instruction comprises any message or signal that indicates to anappliance (e.g., the branch appliance 350 and the central appliance 380)an action to perform with the data. Some examples of the instructionindicate to the appliance to store the data, to retrieve the data, andto forward the data to the computer (e.g., the central servers 370 andthe computers 340). The instruction may be explicit, and implicit basedon instructions indicating to store or retrieve the data. In someembodiments, the instruction indicates an index within a database forstoring and retrieving the data.

The central appliance 380 comprises hardware and/or software elementsconfigured to receive data, determine whether a portion of the data islocally accessible to an appliance (e.g., the branch appliance 350),generate an instruction based on the determination, and transfer theinstruction to the appliance. The central appliance 380 also compriseshardware and/or software elements configured to receive an instructionfrom an appliance (e.g., the branch appliance 350), process theinstruction to obtain the data, and transfer the data to a computer(e.g., the central servers 370). One example of the central appliance380 is described below with respect to FIG. 9. The operations of thecentral appliance 380 are discussed in further detail below in FIGS. 4,5, 6, and 7A-7B.

As illustrated, the branch appliance 350 is configured in-line (orserially) between the computers 340 and the router 360. The centralappliance 380 is also configured serially between the central server 370and the router 390. The branch appliance 350 and the central appliance380 transparently intercept network traffic between the computers 340and the central servers 370. For example, the central appliance 380transparently intercepts data sent from the central servers 370 andaddressed to the computers 340. The computers 340 and the centralservers 370 advantageously require no additional configuration becausethe branch appliance 350 and the central appliance 380 operatetransparently.

Alternatively, the branch appliance 350 and the central appliance 380are configured as an additional router or gateway. As a router, forexample, the branch appliance 350 appears to the computers 340 as anextra hop before the router 360. In some embodiments, the branchappliance 350 and the central appliance 380 provide redundant routing orpeer routing with the router 360 and the router 390. Additionally, inthe bridge and router configurations, the branch appliance 350 and thecentral appliance 380 provide failure mechanisms, such as, fail-to-open(e.g., no data access) or fail-to-wire (e.g., a direct connection to therouter 360).

It will be understood that the branch appliance 350 and the centralappliance 380 perform bi-directional communication. For example, datasent to the branch appliance 350 from the central appliance 380 may bestored in a location locally accessible to the central appliance 380 andin a location locally accessible to the branch appliance 350. If thedata is to be transferred again from the central appliance 380 to thebranch appliance 350, the central appliance 380 may determine that thedata is locally accessible to the branch appliance 350 and generate aninstruction to the branch appliance 350 to retrieve the data. Thecentral appliance 380 transfers the instruction to the branch appliance350 and the branch appliance 350 processes the instruction to obtain thedata. If later, the branch appliance 350 is to transfer the entire databack to the central appliance 380, the branch appliance 350 may use thefact that the central appliance 380 has before transferred the data tothe branch appliance 350. The branch appliance 350 therefore determinesthat the data is locally accessible to the central appliance 380 andgenerates an instruction to the central appliance 380 to retrieve thedata. The branch appliance 350 transmits the instruction to the centralappliance 380 and the central appliance 380 processes the instruction toobtain the data. Therefore, an appliance (e.g., the branch appliance 350and the central appliance 380) in the network memory system 300advantageously uses data transferred to and from the appliance to reducenetwork traffic with other appliances in the network memory system 300.

The network memory system 300 advantageously provides increasedproductivity, reduced IT costs, and enhanced data integrity andcompliance. For example, the network memory system 300 achieves thesimple administration of centralized server systems whereby the centralservers 370 store the primary copy of the data. The network memorysystem 300 improves application performance and data access in thebranch office 310 and central office 320 because not every response to adata request travels over the communication network 330 from the centralservers 370. The branch appliance 350 and the central appliance 380 alsostore to and retrieve from a local copy of the data for subsequentexchanges of the data.

Additionally, the network memory system 300 does not cache the data inthe traditional sense. The data may be retrieved locally even if the URLor filename for the data is different because the data may be identifiedby a pattern for the data itself and not by the URL or filename.Furthermore, unlike web caching, the network memory system 300 ensuresthat the data is coherent by forwarding messages (e.g., data requestsand responses) between the computers 340 and the central servers 370.For example, web caching operates by locally intercepting messages foran authoritative source (e.g., a web server) and responding to themessages such that the web server potentially never sees the messages.In some cases, particularly with dynamic content, the locally cachedcopy may be stale or out-of-date. Advantageously, the network memorysystem 300 provides the data coherency and up-to-date data by thetransparent operation of the network memory system 300 and the principlein which messages are transferred end-to-end (e.g., from the computer340 to the central servers 370), even though the messages and/or thedata may not traverse the communication network 330.

The network memory system 300 does not have the higher cost ofdistributed server systems because the branch appliance 350 and thecentral appliance 380 provide benefits across all applications anddisplace several distributed devices and caches, particularly inmultiple branch implementations. In some embodiments, the branchappliance 350 and the central appliance 380 provide internal storage fora secondary copy of the data. The network memory system 300 also reducesthe hardware and license costs for the branch office 310 and the centraloffice 320 by eliminating the need for the numerous distributed devices.Further, the network memory system 300 minimizes the securityvulnerabilities and patching activities commonly associated with thedistributed systems. Management of the branch appliance 350 and thecentral appliance 380 is simpler than the management of a remotedistributed server. Unlike remote servers, there is no need to configureuser accounts, permissions, and authentication schemes on the branchappliance 350 and the central appliance 380.

FIG. 4 illustrates a message sequence chart for the network memorysystem 300 where a response data 425 to a data request 410 is notlocally accessible to the branch appliance 350, in an exemplaryimplementation of the invention. In this example, the computer 340transmits the data request 410 through the branch appliance 350 and thecentral appliance 380 to the central server 370. Some examples of thedata request 410 are requests for an email attachment, a file, a webpage, and a database query.

In sequence 415, the central servers 370 process the data request 410,and in sequence 420, the central servers 370 generate the response data425 based on the data request 410. Some examples of the response data425 are an email message and attachment, a file, a web page, and resultsretrieved or obtained from the database query. The central servers 370then transmit the response data 425 to the central appliance 380.Alternatively, in some embodiments, the central server 370 addresses theresponse data 425 directly to the computer 340, however, duringtransmission, the central appliance 380 transparently intercepts theresponse data 425. In sequence 430, the central appliance 380 processesthe response data 425 to determine whether a portion of the responsedata 425 is locally accessible to the branch appliance 350.

FIG. 5 illustrates data structures for the network memory system 300 todetermine whether a portion of the data is locally accessible to thebranch appliance 350, in an exemplary implementation of the invention.The data structures include a fine signature hash table (SHT) 505, acoarse signature hash table (SHT) 525, and flow history pages (FHPs)545. The fine SHT 505 includes one or more entries comprising a checkfield 510, a page field 515, and a byte field 520. The coarse SHT 525includes one or more entries comprising a check field 530, a page field535, and a byte field 540. The FHPs 545 include one or more pages (e.g.,page 1-M). Each page (e.g., page N) includes page state information 550and stores data 555.

An appliance of the network memory system 300 (e.g., the branchappliance 350 and the central appliance 380) calculates hashes at everybyte boundary of a data flow (e.g., the response data 425) to be sentacross the communication network 330. In some embodiments, the data flowincludes packets that are in the same Internet Protocol (IP) flow, asdefined by the IP header five tuple of source address, source port,destination address, destination port, and protocol. The hashes may beinfluenced by preceding bytes in the data flow. For example, the hashesare influenced by approximately the n previous bytes, where n determinesthe fingerprint size. Some examples of calculating the hashes arecyclical redundancy checks (CRCs) and checksums over the previous nbytes of the data flow. In some embodiments, rolling implementations ofCRCs and checksums are used where a new byte is added, and a byte from nbytes earlier is removed. To maximize the ability to determine whether aportion of the data flow exists in another appliance in the networkmemory system 300, the hash calculation may span across successive IPpackets in the data flow. In other embodiments, the hash calculationignores patterns that span one or more IP packet boundaries in the dataflow, and the hashes are calculated within a single IP packet.

Each calculated hash is filtered by a fine filter 560 and a coarsefilter 565. The appliance designates the locations in the data flowwhich meet the fine and coarse filter criteria as fine and coarsesync-points, respectively. The fine filter 560 and the coarse filter 565have different filter criteria. Typically, the filter criteria for thecoarse filter 560 is more restrictive and may be used to further filterthose hashes which pass the fine filter. In other words, the fine filterproduces a fine comb of sync-points and the coarse filter produces acoarse comb of sync-points. One example of the filter criteria is thenull filter which allows results in sync-points at all locations. Inanother example, the filter criteria declares a fine sync-point when thetop five bits of the hashes are all zeros and a coarse filter criteriawhich stores or compares hashes when the top ten bits of the hashes areall zeros. The hash at the fine sync-points index the fine SHT 505 andthe hash at the coarse sync-points index the coarse SHT 525. Forexample, the index could be derived from the hash by using a number oflow order bits from the hash. The filter criteria affect the sizing ofthe SHTs 505 and 525 and the probability of matching a hash in the SHTs505 and 525. The more sync-points that are generated the easier repeateddata is identified but, the larger the tables (i.e., the SHTs 505 and525) need to be in order to index a given amount of information for thedata flow. Having a coarse and fine table helps optimize this tradeoff.Alternative implementations may use a single table or multiple tables.

The fine SHT 505 is populated with hashes as the data 555 (e.g., theresponse data 425) is stored and when the data 555 is recalled from diskor other locally accessible storage. The fine SHT 505 finely indexes thedata 555. In some embodiments, the fine SHT 505 holds approximately oneentry for every 100 bytes of the data 555. The coarse SHT 525 ispopulated as the data 555 is stored and is coarsely indexed. Forexample, the coarse SHT 525 may hold one entry for approximately every 4kilobytes (KB) of the data 555. The fine SHT 505 and the coarse SHT 525may be considered short term and long term memory index structures,respectively.

The appliance of the network memory system 300 stores all or part of thecalculated hashes in or compares all or part of the hashes to the checkfield 510 in the SHTs 505 and 525. For example, the central appliance380 verifies a “hit” in the fine SHT 505 by comparing the entirecalculated hash or a number of residual bits of the calculated hash tothe check field 510. If the central appliance 380 finds no matchinghashes in the fine SHT 505 or in the coarse SHT 525, the centralappliance 380 determines that the response data 425 is not locallyaccessible to the branch appliance 350. Each calculated hash for theresponse data 425 in the fine SHT 505 and the coarse SHT 525 is storedor compared depending on the filter criteria for the fine filter 560 andthe coarse filter 565.

The appliance of the network memory system 300 indexes each entry in thefine SHT 505 and the coarse SHT 525 to a page (e.g., by setting the pagefield 515 and the page field 535 to address page N) and byte offset(e.g., by setting the byte field 520 and the byte field 540 to a byteoffset of the data 555) in the FHPs 545. For example, the centralappliance 380 stores the response data 425 in the FHPs 545 at the pagepointed to by the page field 515 and 535 at the byte offset indicated bythe byte field 520 and 540. The byte field 520 of each hash in the fineSHT 505 for the response data 425 points to the start of a finesync-point. The byte field 540 of each hash in the coarse SHT 525 forthe response data 425 points to the start of a coarse sync-point.

In this example, the branch appliance 350 includes a fine SHT 505, acoarse filter 525, and a FHP 545 data structure, and the centralappliance 380 includes a fine SHT 505, a coarse filter 525, and a FHP545 data structure. Each appliance in the network memory system 300maintains the separate data structures, with may include separate filtercriteria for the fine filters 560 and the coarse filters 565. The pagestate information 550, in the FHP 545 of each appliance in the networkmemory system 300, includes page parameters, page ownership permissions,peer state, and a list of valid byte ranges for each appliance in thenetwork memory system 300. The page state information 550 tracks thelocal state of the page (e.g., the FHP 545 in the branch appliance 350,and what parts of the page are used) and the remote state of the page atpeers (e.g., the central appliance 380, and what part of the page in thebranch appliance 350 is used by the central appliance 380).

The branch appliance 350 and the central appliance 380 each write thedata 555 to an assigned page (e.g., the page N or the page N+1) and mayreference a page assigned to another appliance in the network memorysystem 300. Appliances in the network memory system 300 may discover andreconcile the FHPs 545 assigned to other appliances as explained belowwith regard to FIGS. 9 and 10.

Referring again to FIG. 4, the central appliance 380 proceeds with thedetermination that no portion of the response data 425 is locallyaccessible to the branch appliance 350. In sequence 435, the centralappliance 380 generates a store instruction 440. The store instruction440 indicates to the branch appliance 350 to store the response data 425at an index in a database. The central appliance 380 attaches the storeinstruction 440 to the response data 425. The central appliance 380 thentransmits the response data 425 with the store instruction 440 to thebranch appliance 350.

In sequence 445, the branch appliance 350 processes the response data725 with the store instruction 440. In sequence 450, based on the storeinstruction 440, the branch appliance 350 stores the response data 425in the branch appliance 350 at the index within the database. In thisexample, the branch appliance 350 stores the response data 425 in theFHPs 545 at the page and at a particular byte offset indicated by theindex. In sequence 455, the branch appliance 350 forwards the responsedata 425 to the computer 340. As discussed previously, the branchappliance 350 may forward the data to the computer based on explicit andimplicit instructions.

FIG. 6 illustrates a message sequence chart for the network memorysystem 300 where a response data 625 to a data request 610 is locallyaccessible to the branch appliance 350, in an exemplary implementationof the invention. In this example, the computer 340 transmits the datarequest 610 to the central servers 370 through the branch appliance 350and the central appliance 380. In sequence 615, the central servers 370process the data request 610. In sequence 620, the central servers 370generate a response data 625 based on the data request 610. The centralservers 370 then transmit the response data 625 to the central appliance380.

In sequence 630, the central appliance 380 processes the response data625 to determine whether a portion of the response data 625 is locallyaccessible to the branch appliance 350. The central appliance 380 againgenerates hashes for the response data 625, as previously described withrespect to FIGS. 4 and 5. The central appliance 380 filters thegenerated hashes through the fine filter 560 and the coarse filter 565to determine fine and/or coarse sync-points. The central appliance 380looks up the fine sync-points in the fine SHT 505 and the coarsesync-points in the coarse SHT 525. If any of the hashes for the responsedata 625 match (i.e., the check bytes match in either the fine SHT 505and/or the coarse SHT 525), then additional checks (such as directforward comparisons and backward memory comparisons between the responsedata 625 and the data 555 in the FHPs 545) may also be made to determinethe size of the matching region. Further checks using the page stateinformation 550 determine which portion of the response data 625 islocally accessible to the branch appliance 350.

Based on the determination that the entire response data 625 is locallyaccessible to the branch appliance 350, in sequence 635, the centralappliance 380 generates a retrieve instruction 640 that indicates to thebranch appliance 350 to retrieve the response data 625 at an indexwithin the database. The central appliance 380 then transmits only theretrieve instruction 640 to the branch appliance 350. In this manner,the central appliance 380 optimizes network traffic over thecommunication network 330. If the retrieve instruction 640 is smaller insize than the response data 625, the central appliance 380 transmits theretrieve instruction 640. If the retrieve instruction 640 is larger thanthe response data 625, the central appliance 380 transmits the responsedata 625 instead.

In sequence 645, the branch appliance 350 processes the retrieveinstruction 640. In sequence 650, based on the retrieve instruction 640,the branch appliance 350 retrieves the response data 625 at the indexwithin the database. In sequence 655, the branch appliance 350 forwardsthe response data 625 to the computer 340.

FIG. 7A and FIG. 7B illustrate a message sequence chart for the networkmemory system 300 where a portion of a response data 725 to a datarequest 710 is locally accessible to the branch appliance 350, in anexemplary implementation of the invention. The computer 340 transmitsthe data request 710 to the central servers 370 through the branchappliance 350 and the central appliance 380. In sequence 715, thecentral servers 370 process the data request 710. In sequence 720, thecentral servers 370 generate a response data 725 based on the datarequest 710. The central servers 370 then transmit the response data 725to the central appliance 380.

In sequence 730, the central appliance 380 processes the response data725 to determine whether a portion of the response data 725 is locallyaccessible to the branch appliance 350. The central appliance 380computes hashes for the response data 725 and filters the hashes throughthe fine filter 560 and the coarse filter 565 to determine any fine andcoarse sync-points. The central appliance 380 then looks up any finesync-points in the fine SHT 505 and coarse sync-points in the coarse SHT525. In this example, only a portion of the response data 725 is locallyaccessible to the branch appliance 350, meaning that although thecentral appliance 380 finds at least one match in the SHTs 505 and 525,additional checks (such as the direct forward comparison and thebackward memory comparison with the response data 725 and the data 555)determine that only a portion of the response data 725 is locallyaccessible to the branch appliance 350.

The central appliance 380 stores the generated hashes for thenon-locally portion of the response data 725 (otherwise known as thedeltas) in the SHTs 505 and 525, and stores the deltas in the FHPs 545.The central appliance 380 will transmit the deltas (i.e., the portion ofthe response data 725 that is not locally accessible) to the branchappliance 350.

In sequence 735, the central appliance 380 generates retrieve and storeinstructions 740. The retrieve instruction indicates to the branchappliance 350 to retrieve the locally accessible portion of the responsedata 725 at an index within the database. The store instructionindicates to the branch appliance 350 to store the deltas at an indexwithin the database. The store instruction may also indicate to thebranch appliance 350 to store another copy of the portion of theresponse data 725 locally accessible to the branch appliance 350 withthe deltas. The entire response data 725 will then be locally accessiblein the database to the branch appliance 350. The central appliance 380attaches the deltas to the retrieve and store instructions 740. Thecentral appliance 380 then transmits the non-locally accessible portionof the response data 725 with retrieve and store instructions 740 to thebranch appliance 350.

In sequence 745, the branch appliance 350 processes the non-locallyaccessible portion of the response data 725 with retrieve and storeinstructions 740. In sequence 750, based on the retrieve instruction,the branch appliance 350 retrieves the locally accessible portion of theresponse data 725 at the index in the database. In sequence 755, thebranch appliance 350 obtains the response data 725 from the retrievedlocally accessible portion and the transferred deltas (i.e., thetransferred non-locally accessible portion of the response data 725). Insequence 760, based on the store instruction, the branch appliance 350stores the deltas (and potentially the retrieve locally accessibleportion of the response data 725) at the index in the database. Insequence 765, the branch appliance 350 transmits the entire responsedata 725 to the computer 340.

Alternatively, in addition to the examples in FIGS. 4, 5, 6, and 7A-7Billustrating a request for the data originating from the computer 340 tothe central servers 370, the computer 340 may also transmit data to thebranch appliance 350 addressed to the central servers 370. The branchappliance 350 determines whether a portion of the data is locallyaccessible to the central appliance 380. Then, for example, if the datais locally accessible the central appliance 380, the branch appliance350 generates a retrieve instruction indicating to the central appliance380 to retrieve the data and forward the data to the central server 370.

In still further embodiments, the instruction may indicate a pluralityof indexes. Referring again to FIG. 7B, in sequence 750, based on theretrieve instruction indicating a plurality of indexes for the responsedata 725, the branch appliance 350 may retrieve the locally accessibleportion of the response data 725 at different locations based on theplurality of index. For example, the branch appliance 350 may retrieve aportion of the response data 725 from RAM, a portion from a hard disk,and a portion from a NAS device. Similarly, in sequence 760, based onthe store instruction indicating a plurality of indexes for the responsedata 725, the branch appliance 350 may stores the deltas in the databaseand after obtaining the entire response data 725, store the entireresponse data 725 in a different location (e.g., in a different locationin the database, in a disk drive, or in a NAS device) than thepreviously locally accessible portion.

FIG. 8 illustrates a block diagram of the branch appliance 350, in anexemplary implementation of the invention. The branch appliance 350includes a processor 810, a memory 820, a WAN communication interface830, a LAN communication interface 840, and a database 850. A system bus880 links the processor 810, the memory 820, the WAN communicationinterface 830, the LAN communication interface 840, and the database850. Line 860 links the WAN communication interface 830 to the router360 (in FIG. 3). Line 870 links the LAN communication interface 840 tothe computers 340 (in FIG. 3).

The database 850 comprises hardware and/or software elements configuredto store data in an organized format to allow the processor 810 tocreate, modify, and retrieve the data. The database 850 may organize thedata to enable the determination of whether a portion of the data islocally accessible to an appliance, and to enable quick retrieval oflocally accessible data to the branch appliance 350. The hardware and/orsoftware elements of the database 850 may include storage devices, suchas RAM, hard drives, optical drives, flash memory, and magnetic tape. Insome embodiments, the branch appliance 350 implements a virtual memorysystem with linear addresses, the locally accessible data, and the datastructures discussed with respect to FIG. 5 in the database 850.

FIG. 9 illustrates a block diagram of the central appliance 380, in anexemplary implementation of the invention. The central appliance 380includes a processor 910, a memory 920, a WAN communication interface930, a LAN communication interface 940, and a database 950. A system bus980 links the processor 910, the memory 920, the WAN communicationinterface 930, the LAN communication interface 940, and the database950. Line 960 links the WAN communication interface 930 to the router390 (in FIG. 3). Line 970 links the LAN communication interface 940 tothe central servers 370 (in FIG. 3). In some embodiments, the branchappliance 350 and the central appliance 380 comprise the identicalhardware and/or software elements. Alternatively, in other embodiments,the central appliance 380 may include hardware and/or software elementsproviding additionally processing, communication, and storage capacity.

Advantageously, the network memory system 300 improves applicationperformance and data access. In some embodiments, by storing a secondarycopy of the data locally accessible to the branch appliance 350 and thecentral appliance 380, the network memory system 300 minimizes theeffects of latency and reduces network traffic over the communicationnetwork 330 to the central servers 370. Additionally, while the centralservers 370 maintain the primary copy of the data, the central servers370 potentially do not transfer the actual data over the communicationnetwork 330 for every request/response. Furthermore, accelerated accessto the data locally accessible to the branch appliance 350 and thecentral appliance 380 is not limited to a particular application or datacontext.

In some embodiments, the network memory system 300 includes a securetunnel between the branch appliance 350 and the central appliance 380.The secure tunnel provides encryption (e.g., IPsec) and access controllists (ACLs). Additionally, in other embodiments, the secure tunnelincludes compression, such as header and payload compression. The securetunnel may provide fragmentation/coalescing optimizations along witherror detection and correction.

FIG. 10 illustrates a network memory system 1000 between a first office1010, a second office 1030, and a third office 1060, in an exemplaryimplementation of the invention. The first office 1010 includes acomputer 1015 and a first network memory appliance (NMA) 1020. Thesecond office 1030 includes a computer 1040 and a second NMA 1050. Thethird office 1060 includes a third NMA 1070 and a server 1080. The firstoffice 1010 is linked to the second office 1030 and the third office1060, (e.g., through routers not shown). The second office 1030 is alsolinked the third office 1060.

The first NMA 1020, the second NMA 1050, and the third NMA 1070 comprisehardware and/or software elements, similar to the branch appliance 350and the central appliance 380, configured to receive data, determinewhether the data is locally accessible to an appliance, generate aninstruction based on the determination, and transfer the instruction tothe appliance. The first NMA 1020, the second NMA 1050, and the thirdNMA 1070 also comprise hardware and/or software elements configured toreceive an instruction from an appliance, process the instruction toobtain data, and transfer the data to a computer.

Advantageously, in this multi-office example, the network memory system1000 provides for locally accessible data in each office. The first NMA1020, the second NMA 1050, and the third NMA 1070 receive data,potentially destined for a computer and/or server in another office, anddetermine whether a portion of the data is locally accessible to an NMAin that office. To further enhance operation and the exchange of databetween the first NMA 1020, the second NMA 1050, and the third NMA 1070,each NMA performs a discovery and reconciliation. During discovery andreconciliation the virtual memory map of the network memory system 1000is updated. For example, each NMA updates the pages of the FHPs 545 inthe NMA with references for data locally accessible in the networkmemory system 1000 and to which NMA the data is locally accessible.

FIG. 11 illustrates a message sequence chart for the network memorysystem 1000 for discovery and reconciliation, in an exemplaryimplementation of the invention. In this example, the computer 1015 inthe first office 1010 transmits data to the first NMA 1020 for the firsttime addressed to the computer 1040 in the second office 1030. The firstNMA 1020 transmits the data with a store instruction to the second NMA1050 indicating to store the data in a database in the second NMA 1050.In sequence 1110, the second NMA 1050 stores the data in the database,and the second NMA 1050 transmits the data to the computer 1040.

The computer 1015 in the first office 1010 then transmits the same datato the first NMA 1020 addressed for the first time to the server 1080 inthe third office 1060. The first NMA 1020 transmits the data with astore instruction to the third NMA 1070 indicating to store the data ina database in the third NMA 1070. In the sequence 1115, the third NMA1070 stores the data in the database, and the third NMA 1070 transmitsthe data to the server 1080.

In sequence 1120, 1125, and 1130, the first NMA 1020, the second NMA1050, the third NMA 1070 perform discovery and reconciliation includingupdate the virtual memory map. In this example, the first NMA 1020, thesecond NMA 1050, and the third NMA 1070 exchange information (e.g., thepage state information 550) about which parts of the FHPs 545 each NMAhas available locally. For example, to update the FHPs 545 in the secondNMA 1050, the second NMA 1050 performs a discovery and reconciliationwith the first NMA 1020 and the third NMA 1070. Similarly, each NMAperforms discovery and reconciliation with every other peer NMA.

During the discovery and reconciliation between the second NMA 1050 andthe first NMA 1020, for example, the second NMA 1050 discovers from thefirst NMA 1020 that the data (transmitted from the computer 1015 to thecomputer 1040 and the server 1080) is locally accessible to the thirdNMA 1070. The FHPs 545 of the first NMA 1020 include references to thedata (e.g., in the page state information 550) and because the first NMA1020 transferred the data to the third NMA 1070, the FHPs 545 indicatethat the data is locally accessible to the third NMA 1070. The secondNMA 1050 reconciles the references for the data in the FHPs 545 andfurther indicates that data is locally accessible to the third NMA 1070.

Referring again to FIG. 11, in sequence 1135, after the discovery andreconciliation in sequences 1120, 1125, and 1130, the computer 1040 inthe second office 1030 transmits the data addressed to the server 1080in the third office 1060. The data is intercepted by the second NMA1050, and in sequence 1140, the second NMA 1050 determines whether aportion of the data is locally accessible to the third NMA 1070. Sincethe discovery and reconciliation, the FHPs 545 in the second NMA 1050indicates that data is locally accessible to the third NMA 1070. Insequence 1145, the second NMA 1050 generates a retrieve instructionindicating to the third NMA 1070 to retrieve the data from an indexwithin the database. The second NMA 1050 transfers the retrieveinstruction to the third NMA 1070.

In sequence 1150, the third NMA 1070 processes the retrieve instruction.In sequence 1155, based on the retrieve instruction, the third NMA 1070retrieves the data at the index within the database. In sequence 1160,the third NMA 1070 forwards the data to the server 1080.

Therefore, the network memory system 1000 provides improved applicationperformance and data access between the first office 1010, the secondoffice 1030, and the third office 1060. The network memory system 1000provides access to locally accessible data, similar to distributedservers, without the complex management involved with replication andsynchronization of the data among the distributed servers. Additionally,the network memory system 1000 reduces network traffic between theoffices. Furthermore, discovery and reconciliation provides performanceadvantages by periodically or continuously updating the FHPs 545 in eachNMA.

The above-described functions can be comprised of executableinstructions that are stored on storage media. The executableinstructions can be retrieved and executed by a processor. Some examplesof executable instructions are software, program code, and firmware.Some examples of storage media are memory devices, tape, disks,integrated circuits, and servers. The executable instructions areoperational when executed by the processor to direct the processor tooperate in accord with the invention. Those skilled in the art arefamiliar with executable instructions, processor(s), and storage media.

The above description is illustrative and not restrictive. Manyvariations of the invention will become apparent to those of skill inthe art upon review of this disclosure. The scope of the inventionshould, therefore, be determined not with reference to the abovedescription, but instead should be determined with reference to theappended claims along with their full scope of equivalents.

1. A network memory system comprising: a first appliance configured to receive data, determine whether a portion of the data is locally accessible to a second appliance, generate an instruction based on the determination, and transfer the instruction to the second appliance over a communication network; and the second appliance configured to receive the instruction from the first appliance over the communication network, process the instruction to obtain the data, and transfer the data to a computer.
 2. The network memory system of claim 1 wherein the communication network comprises a wide area network.
 3. The network memory system of claim 1 wherein the first appliance is further configured to generate a plurality of instructions based on the determination.
 4. The network memory system of claim 3 wherein the plurality of instructions indicate a plurality of indexes for retrieving the data.
 5. The network memory system of claim 1 further comprising: the second appliance configured to receive the data, determine whether a portion of the data is locally accessible to the first appliance, generate another instruction based on the determination, and transfer the another instruction to the first appliance over the communication network; and the first appliance configured to receive the another instruction from the second appliance over the communication network, process the another instruction to obtain the data, and transfer the data to another computer.
 6. The network memory system of claim 1 further comprising: the first appliance configured to determine whether the data is locally accessible to a third appliance, generate another instruction to the third appliance based on the determination, and transfer the another instruction to the third appliance over the communication network; and the third appliance configured to receive the another instruction from the first appliance over the communication network, process the another instruction to obtain the data, and transfer the data to another computer.
 7. The network memory system of claim 1 wherein the instruction indicates to store the data in a database.
 8. The network memory system of claim 7 wherein the second appliance is further configured to store the data in the database based on the instruction.
 9. The network memory system of claim 1 wherein the instruction indicates to retrieve the data from a database.
 10. The network memory system of claim 9 wherein the second appliance is further configured to retrieve the data from the database based on the instruction.
 11. The network memory system of claim 1 wherein the first appliance is further configured to transfer to the second appliance a portion of the data that is not locally accessible, and the second appliance is further configured to receive the transferred portion of the data, retrieve the portion of the data that is locally accessible to the second appliance, and process the portions to obtain the data.
 12. The network memory system of claim 1 wherein the instruction indicates an index within a database for the data.
 13. The network memory system of claim 1 wherein the first appliance is further configured to receive the data from a computer server.
 14. The network memory system of claim 1 wherein the second appliance is further configured to transfer the data to the computer based on a request for the data over the communication network by the computer.
 15. A method for network memory comprising: receiving data in a first appliance; determining in the first appliance whether a portion of the data is locally accessible to a second appliance; generating in the first appliance an instruction based on the determination; transferring the instruction to the second appliance over a communication network; receiving the instruction in the second appliance over the communication network; processing in second appliance the instruction to obtain the data; and transferring the data to a computer.
 16. The method for network memory of claim 15 wherein the communication network comprises a wide area network.
 17. The method for network memory of claim 15 further comprising generating in the first appliance a plurality of instructions based on the determination.
 18. The method for network memory of claim 17 wherein the plurality of instructions indicate a plurality of indexes for retrieving the data.
 19. The method for network memory of claim 15, further comprising: receiving the data in the second appliance; determining in the second appliance whether a portion of the data is locally accessible to the first appliance; generating in the second appliance another instruction based on the determination; transferring the another instruction to the first appliance over the communication network; receiving the another instruction in the first appliance over the communication network; processing in first appliance the another instruction to obtain the data; and transferring the data to another computer.
 20. The method for network memory of claim 15, further comprising: determining in the first appliance whether the data is locally accessible to a third appliance; generating in the first appliance another instruction to the third appliance based on the determination; transferring the another instruction to the third appliance over the communication network; receiving in the third appliance the another instruction over the communication network; processing in the third appliance the another instruction to obtain the data; and transferring the data to another computer.
 21. The method for network memory of claim 15 wherein the instruction indicates to store the data in a database.
 22. The method for network memory of claim 21, further comprising storing the data by the second appliance in the database based on the instruction.
 23. The method for network memory of claim 15 wherein the instruction indicates to retrieve the data from a database.
 24. The method for network memory of claim 23, further comprising retrieving the data by the second appliance from the database based on the instruction.
 25. The method for network memory of claim 15, further comprising: transferring to the second appliance a portion of the data that is not locally accessible; receiving by the second appliance the transferred portion of the data; retrieving by the second appliance the portion of the data that is locally accessible; and processing in the second appliance the portions to obtain the data.
 26. The method for network memory of claim 15 wherein the instruction indicates an index within a database for the data.
 27. The method for network memory of claim 15 wherein receiving the data in the first appliance comprises receiving the data from a computer server.
 28. The method for network memory of claim 15 wherein transferring the data to the computer comprises transferring the data based on a request for the data over the communication network by the computer.
 29. A network memory system comprising: means for receiving data in a first appliance; means for determining in the first appliance whether a portion of the data is locally accessible to a second appliance; means for generating in the first appliance an instruction based on the determination; means for transferring the instruction to the second appliance over a communication network; means for receiving the instruction in the second appliance over the communication network; means for processing in second appliance the instruction to obtain the data; and means for transferring the data from the second appliance to a computer.
 30. The network memory system of claim 29 wherein the communication network comprises a wide area network.
 31. The network memory system of claim 29 wherein the instruction indicates to store the data in a database.
 32. The network memory system of claim 29 wherein the instruction indicates to retrieve the data from a database.
 33. The network memory system of claim 29, further comprising: means for transferring the data a portion of the data is not locally accessible; means for receiving in the second appliance the transferred portion of the data; means for retrieving by the second appliance the portion of the data that is locally accessible; and means for processing in the second appliance the portions to obtain the data.
 34. The network memory system of claim 29 wherein the instruction indicates an index within a database for the data. 